II. RESEARCH PLAN

	A. THE PILOT ASSEMBLY STATION

The basic goal of the proposed project  is  to  implement  a  general
purpose  assembly  station  and  to  prove  its applicability to real
problems. The project will be considered a success  if  one  or  more
enterprises  chooses  to  manufacture  and market the general purpose
station. There are a number of hardware and software problems  to  be
solved in meeting our goal.

In one configuration, the assembly station is assumed to be connected
to a large time-shared computer facility. Not only does  this  appear
to  be a good design for our problem, but it is also necessary for an
integrated  programmed  automation  facility.  The  only   emphasized
statement  in  the ARPA study report [Anderson 1972] is: "The maximum
impact  of  computers  in  manufacturing  will  come  from  complete,
real-time  computer  cognizance  and  control  of  all  processes and
resources, allowing the precise scheduling and  allocation  of  these
resources."

Each  assembly  station will consist of a small digital computer, one
or more manipulators and cameras. Manipulators and  cameras  will  be
combined  as  modules  in  systems  to suit a given task, and will be
digitally controlled by the station computer.  The  station  computer
may be connected to a central computer or may stand alone. The latter
configuration is less efficient in  the  long  run,  but  allows  for
incremental automation.

The program in the station computer will be flexible as to the number
and position of manipulators and cameras. It will control input  from
the  cameras,  provide for some pre-processing of the picture and the
application of simple tests to the  visual  data.   In  the  case  of
manipulators  it will provide for simultaneous servo control. It will
also process  other  inputs,  such  as  from  touch  sensors  on  the
manipulators,  from photocell interrupt devices, limit switches, etc.
Both analog and binary outputs will be available for the  control  of
other devices.

Each  station  computer will have access to a task program specifying
the actions and tests it should perform to carry out its  task.  This
task  program  will  be compiled by the central computer. Compilation
will consist of the analysis of predicted scenes in order to  specify
simple tests that the station computer can carry out, the calculation
of collision free trajectories for the manipulators, the sequence  of
operations  and  simple  contingency  plans. etc. We are planning for
tasks  whose  normal  execution  does  not  require  detailed   scene
analysis.  If  scene analysis becomes necessary (e.g. a dropped part)
we envision the station sending an image to the central computer  for
processing.

The  central  computer will perform overall control of the individual
assembly stations and the relative rates of  production.  If  at  any
station  a  planned test fails, for which a contingency plan does not
exist, or any unexpected sensor signals  are  received,  the  central
computer  will  also be interrupted and appropriate actions taken. If
the assembly station is not connected to a  central  computer,  these
operations will require human intervention.


Most of the hardware required  for  the  assembly  station  has  been
developed  for  the  laboratory.  Manipulators  and  cameras  will be
discussed in detail below. The major additional problem is to  refine
the   existing   designs   to  meet  the  demands  of  an  industrial
environment.

We are now acquiring small computer  such  as  the  PDP-11  and  will
interface  the existing arms to it together with one or more cameras.
A software package for the small computer will then be  developed  in
order to control the manipulators, read the TV cameras and to perform
low level visual analysis. Programs  to  plan  tasks,  calibrate  the
system  and  to  perform higher level visual analysis, error recovery
and strategy planning will be written in a high  level  language  for
the time-sharing processor.

The most difficult part of the assembly station, and the heart of the
project is a demonstration of the feasibility of the station. We plan
to do this by developing programs which actually assemble and inspect
a variety of fairly complex manufactured items.

A typical example is our current effort, assembly  of  an  automobile
water  pump.  The  major  parts  are  to  be  located, the screws are
supplied by a feeder, as is standard. Locating the body and cover  of
the  pump  require  a gross localization, then a fine localization of
screw  holes.  Touch  provides  a  final  accuracy  and   independent
coordination  of the hand. The placing of alignment pins within screw
holes could be monitored visually; although in this case,the previous
actions   provide  adequate  accuracy.  Placing  of  the  cover  over
alignment pins can be carried  out  without  vision.  Improved  force
sensing now under development will be useful for that operation.

As  we move to more difficult tasks, the sophistication required goes
up quite rapidly.  The  current  manipulation  routines  seem  to  be
expandable  to cover anticipated problems, especially since we intend
to employ special purpose manipulators (tools) for  some  tasks.  The
main part of our effort will be vision for assembly; an assembly task
will be performed using vision, touch, and  two  hands.   The  vision
effort  will  be  integrated with other abilities, and while no great
generality will be stressed, new modules will  be  incorporated  into
the  system,  and  a set of techniques will evolve. These will depend
heavily on techniques similar to  those  of  our  recent  studies  in
visual  feedback  and hand/eye coordination [Gill 1972]. The internal
model of the world will be  used  to  predict  visual  appearance  of
selected  areas  of  the  scene, and a set of verification procedures
will confirm or contradict the predictions. Many  of  the  operations
will  be  connected  with  supplying  feedback  to  manipulation, for
alignment and insertion. These visual procedures will have models for
the parts and approximate locations.

This and other visual tasks connected with assembly will initially be
performed as special  case  problems.  In  addition,  to  make  these
techniques   available,   we  will  pursue  development  of  strategy
programs. This step is essential if intelligent assembly devices  are
to  be conveniently programmed for complex tasks. Within the division
of labor between large and small machines, this facility must  reside
in  the  central  time-sharing  facility which has large memory, file
storage, and problem-solving languages. The system must have a  model
of  the  parts  for  assembly,and  a  sequence  of  instructions  for
assembly. It will be necessary to have a representation of shape  and
models for usual assembly operations such as insertion and alignment,
and the ability to solve simple problems  in  connection  with  these
operations,  all  this  to  enable  convenient communication with the
user. When these facilities exist, the addition of new  features  and
operations  can be incorporated smoothly, so that users can call on a
growing library of planning modules. The result of a planning  effort
is  a  cycle  of  machine  planning and man-machine interaction which
results in a plan  network  for  the  mini  computer  which  controls
individual machines.


	B.  RESEARCH IN VISION

Research in computer vision has progressed so that the larger part of
our  effort is directed toward vision in real scenes. We outline past
research, including work on visual feedback,  development  of  useful
modules,  analysis  of  some outdoor scenes, interpretation of scenes
with curved objects, and a study of sensor systems.

Our  past  work on visual feedback [Gill 1972] in stacking blocks and
inserting objects  in  holes  is  directly  relevant  to  vision  for
assembly.  The  issues  involved  are:  coordinating  hand  and  eye,
utilizing incremental accuracy of vision and  manipulation,  accurate
prediction  to  limit  search,  and  limiting accumulation of errors.
Visual guidance in picking  up  objects  and  stacking  or  inserting
objects  overcomes  errors  in the visual and arm coordinate systems,
and errors in correspondence between the two coordinate systems. This
was  done under realistic conditions, with the hand still holding the
object and in view; the system predicts the appearance of  edges  and
corners  within  a  small  window around the edges to be aligned, and
uses a corner/edge finder to determine alignment error. The arm makes
accurate  incremental  error  correction.  If predictions can be made
accurately, the search window can be small, which increases the speed
of  response  and decreases possibility of confusion with other areas
of the scene. A major limitation to accuracy of  prediction  (and  to
long-term  reliability  in  the  industrial  environment) is drift of
calibration.  That  work   incorporates   calibration   updating   by
self-calibration  to  coordinate  hand and eye coordinate systems and
provide for reliability, as well as to  limit  search  by  accurately
predicting  expected  features. Visual feedback with stereo vision is
much more powerful than without; we have demonstrated  stereo  visual
feedback in a program [Binford unpublished].

We  have  a  set of visual modules to be used in a goal-directed way.
These modules  answer  questions  which  arise  in  practical  visual
feedback  tasks,  and  in analysis of outdoor scenes. Are there color
edges in a selected window? Are there edges in  brightness?  What  is
the  color  of  the  region  in the window? Several preliminary color
region analysis programs have been developed  [Binford,  unpublished,
Bajcsy  1972].  An  edge  operator  [Hueckel 1969] and a region-based
corner/edge finder [Gill 1972] both operate on small windows  of  the
scene,  as  directed  by  a higher level. An edge verifier is used to
confirm or reject edges  predicted  from  context  [Tenenbaum  1970].
Color  identification has long been demonstrated. Recently, a program
has been implemented [Thomas unpublished]  based  on  Land's  Retinex
model  of  color  constancy in vision, which defines a color triangle
from  ordering  of  brightnesses.  The  center  of  the  triangle  is
designated  as  white. The program shows a color constancy similar to
humans; table top scenes are assigned colors reasonably well under  a
variety  of  lighting  with different spectra: sunlight, flourescent,
and incandescent.

Understanding outdoor  scenes  is  a  more  complex  task  than  many
industrial  visual  operations,  but  it  has  utility  for planetary
exploration; it is also  a  useful  exercise  for  developing  color,
texture, and stereo operations. Bajcsy analyzed outdoor scenes with a
combination of color and texture region analysis [Bajcsy 1972]; depth
cues  were taken from the gradient of texture. An attempt was made to
obtain  consistent  interpretations  by  making   three   dimensional
interpretations, and confirming boundaries by independent properties.
Natural objects and natural scenes have been used for our  work  with
motion    parallax    [Nevatia    in   draft]   and   stereo   vision
[Quam,Hannah].These are complementary techniques of depth  perception
which  are applicable for planetary exploration and industrial tasks.
Motion  parallax  can  be  used  for  planning  of   navigation   and
characterizing  large scale terrain features, since it provides large
baselines.

Another aspect  of  vision  with  direct  application  to  industrial
enviroments  is  explored  in  our  work with curved objects. Current
programs use depth information to make useful descriptions of objects
like  a  doll,  a toy horse, a snake, or a glove [Agin]. Programs can
segment these objects into parts like legs, arms or fingers, and link
these   parts   together  to  form  descriptions  of  entire  objects
[Nevatia]. These operations are precursors to a recognition  process.
The  work  depends on a new representation of shape [Binford 1971] in
terms of volume components and a part/whole  segmentation.  Primitive
parts  are  described  by  an axis which is a space curve, and normal
cross sections along the axis.  The  representation  is  based  on  a
formalism,   "generalized   translational   invariance".   There  are
significant theoretical reasons why this representation  leads  to  a
natural choice of parts by simple procedures, in distinction to other
suggested  representations.  We  developed  a  laser  ranging  device
[Binford  1970,Agin]  which  was  operational  two  years  ago. It is
particularly interesting that the device adds a cost  of  only  about
$400 to the cost of a visual system.

A study of visual and depth ranging sensor has been carried out which
provides a background for choosing sensors and interface, and  making
cost  estimates  [Binford  1972, Earnest 1967].  We have maintained a
calibrated hand and vision system for several years [Sobel 1970], and
have  developed  self-calibration  techniques  with  one and two eyes
[Yakimovsky unpublished, Hannah 1972, Gill 1972].

A system for understanding scenes of blocks  from  single  views  has
been  implemented  [Falk  1970]; it segments objects, matches against
internal  models,  predicts  appearance  from  its  assumptions,  and
searches  for  verification  and contradiction based on the predicted
image. Support and spatial relations are  also  obtained.  There  has
been  past  work  on  programs  to  obtain  line drawings as input to
recognition systems. An earlier system was based on  limited  context
of expecting continuation of edges at vertices and predicting missing
edges in certain expected configurations [Grape 1970]. A more  recent
approach has been to use models within the process of generating line
drawings, so that the most efficient  next  guess  can  be  made  and
context used intrinsically in the formation of line drawings.

VISION FOR ASSEMBLY

This  is  a  key part of the proposed project. We propose to evaluate
and implement visual facilities  required  by  a  robot  with  simple
vision, in conjunction with force and touch. These facilities must be
modularized by working out several examples, and made fast enough and
small  enough to be practical for a minicomputer-based robot. We have
considerable experience with such standardization,  and  with  visual
feedback  techniques  in  the domain of blocks. We are now at work on
extending visual feedback techniques to typical  assembly  tasks.  We
are  also  evaluating  methods of automating visual feedback. We have
well-developed systems for modeling objects from the console  and  to
predict  their  appearance  using  techniques  from computer graphics
[Baumgart 1972a,b]. We can predict curved edges from our  models.  We
will use simple region analysis programs and edge operators to locate
curved edges  predicted  within  a  small  window.  Automated  visual
feedback   would  combine  with  evolving  strategy  programs  in  an
interactive system, Assembly Robot Management System, ARMS.  We  will
maintain a calibrated system and improve calibration updating so that
reliability is maintained at a high level.

We have developed a special case program for locating the  pump  body
in  the  pump assembly and moving it to the standard location. We are
at work on several other such tasks involved  in  the  pump  assembly
which  contain  components  of  general  utility for visual feedback.
These tasks are typical of the simple visual operations which we will
implement or adapt.

VISION RESEARCH

While  continuing  research activities are not to be supported by the
proposed project, in all likelihood, the research will suggest useful
programs for the assembly system, particularly work in recognition of
complex objects using depth data.   The  computer  now  has  symbolic
internal  models  of  objects.  We are now in a position to recognize
objects such as a glove, a hammer, etc.   We  will  immediately  deal
with matching descriptions for recognition.  The important extensions
we wish to deal with are occlusion  and  access  to  a  large  visual
memory.   These  will  involve  matching  of partial descriptions and
encoding of partial descriptions as  indices  to  a  small  class  of
similar  shapes,  which  can then be compared in more detail.  We are
working toward making the hardware stable enough that it can maintain
calibration,  and  software streamlined so that the depth measurement
system can be integrated with other hand/eye facilities.   We  intend
to  apply large parts of our programs for description and recognition
of curved objects using only monocular intensity information. This is
a  much harder problem, but within our representation, shares much of
the same code, and relies on  the  same  conceptual  basis  of  local
three-dimensional models.


There is a growing  effort  on  navigation  of  vehicles  in  outdoor
scenes.  While  this  is  not  directly  relevant  to  the industrial
environment, it does  reflect  the  growing  involvement  in  complex
visual  tasks,  and will lead to further development of an integrated
visual system. One approach which has already given  initial  results
[Yakimovsky] is to combine region growing with interpretation using a
world model.  Thus, regions are combined not simply on the  basis  of
uniformity but because combining them improves the interpretation. At
the same time, we will be combining more facilities into these visual
systems, and more usable knowledge in the world model.

	C. MANIPULATOR DEVELOPMENT

Manipulator   research  began  at  Stanford  Artificial  Intelligence
Laboratory  in  1965  with  the  acquisition  and  programming  of  a
prototype  Rancho Orthotic Arm.  A later version of this arm was used
in vision directed block stacking programs.

Our present arm, the result of a continuing design project,  replaced
the  Rancho Arm in 1970. This arm like its predecessor was controlled
by a digital servo program running under the time  sharing  computer.
The new arm was used in the "Instant Insanity" demonstration in 1971.
A second arm of the same type,  with  only  minor  modifications,  is
currently  being interfaced to the computer.   Both arms will share a
common work space and will  be  able  to  be  used  individually,  or
together for two handed tasks.

Each  joint  of the arm is calibrated in terms of torque versus drive
level and it is possible to predict the joint drive levels to produce
a  given  force at the hand. Further, for the hand to exhibit degrees
of freedom of motion, various joints are selected to be "free",  that
is  to have no position feedback applied to them. By exerting a force
at the hand and providing that certain degrees of freedom exist it is
possible  to  perform  a  much  wider  class  of  tasks  than  simple
positional control, for instance screwing in  screws,  mating  parts,
inserting tools and turning cranks.

Touch  sensors on the fingers allow the hand to sense objects without
disturbing them.   When activated the arm will stop when any  of  the
sensors  touch.  Branching arm instructions can test the state of the
sensors and modify the execution of an arm program. When  screwing  a
nut  onto  a  bolt, for instance, the hand is first centered over the
bolt by touching it.

Control of the arms is by a series of high level instructions;  these
instructions  may  be executed directly, or may be assembled together
into an arm program. Arm programs may then be executed,  as  desired,
by  the  control  program.   The control instructions may be accessed
directly by other programs, using the message procedure mechanism, or
in the form of symbolic source file.

Positional  accuracy  at  the  hand  is  typically  0.1  inches  with
repeatability of 0.02 inches. Force control at the hand is within  10
ounces. The data rate to run the arm from an assembled arm program is
of the order of 100 bits/second.

Manipulator research moves  ahead  simultaneously  on  three  fronts:
control,  primitives  and  devices.      Tasks are translated into an
existing hand language and where the task cannot be  accomplished  in
terms  of  existing  primitives,  new  primitives are formulated in a
consistent manner.   When a new primative is  formulated,  real  time
routines  are  developed  to perform the action.  This also leads, in
many cases, to the development of new  hardware.        Also  as  new
developments  occur  in  hardware design, existing real time routines
are either re-written or modified.

Research is now concentrating on the following areas:

Strain gauge instrumentation of the wrist, in order to allow for more
sensitive  interaction  of  the  hand with a task.   High resolution,
high sensitivity touch sensors, to increase tactile  recognition  and
control  abilities.      These  developments will lead in turn to new
control concepts.

Development of tools for the hand. These tools  can  be  instrumented
such  that  the  computer can feel and or see at the end of the tool.
This should be a very important area of research as  it  goes  beyond
present human capabilities.

Two handed arm control, the planning and execution of tasks requiring
the coordinated motion of both arms. This will require  an  extension
of  the  existing  servo  program and the analysis of synchronization
problems. The development of collision avoiders, so  that  arms  will
not collide with other objects and with each other.

Modification  of  the  existing  arm  design  to  provide  for better
incremental motions, more degrees of freedom and a more rugged design
suitable  for  industrial  use.   Development of arm and eye computer
interfaces.   These contain  synchronized  analog/digital  converters
for input and current controlled drivers for output.

The  development  of  new hands is expected as more complex tasks are
attempted.   At present, with only one hand, we have been  performing
the  positioning  type  of  hand  actions.    When  the second arm is
operational we will be able to perform holding and pulling actions as
well.

	D.  SYSTEM REQUIREMENTS FOR ADVANCED AUTOMATION

A  key  element in the development of any large programming effort is
the software support.  There has been continual  effort  along  these
lines  at  the  Stanford  Artificial Intelligence Laboratory and many
useful ideas have been developed.  We propose to extend the  research
along  two  separate lines:  support for research and system software
for operating automated systems.  We will briefly describe  our  past
efforts in this area and what we propose to undertake.

The  Stanford  Artificial  Intelligence  Laboratory  has pioneered in
several areas of system programming, including  timesharing  sys36700
interactive graphics, and programming languages.

A  great  deal  of  additional effort has gone into direct support of
hand-eye research.   Very early, we established  a  set  of  standard
formats  and  library routines [Pingle 72a].  A programming language,
SAIL[Swinehart], was implemented in 1968.  The language  incorporates
special   features  for  large  real-world-oriented  problem  solving
systems  and  is  currently  in  use  at  a  number   of   Artificial
Intelligence and other laboratories.

As  we began to assemble larger and more sophisticated collections of
programs, we felt a need  for  flexible  mechanisms  for  cooperating
sequential  processes.   The  resulting monitor and language features
[Feldman & Sproull 1971] have  worked  well  for  us  and  have  been
influential   on  other  projects.     Recently,  we  have  added  an
additional set of features to SAIL  to  facilitate  non-deterministic
and  event-directed  programming[Feldman 1972].   The early reactions
to this work, both within the laboratory and without  has  been  very
encouraging.

We  expect  relatively  little  work  to be needed on further systems
support for basic research.   There will undoubtedly  be  efforts  to
take  advantage of new hardware and of new programming ideas that may
develop in the course of this program.    Almost  all  of  this  work
should be covered by other funding.

The  development  of  software  for the assembly station will require
more  of  our  effort.    The  central-local  computer  configuration
requires  code to be resident in the local computer and in the shared
machine.    The low level programs in the local machine will  not  be
modified  often,  thus  we can assume that a compiler resident in the
central machine will suffice.   The  planning  programs  and  general
vision programs will be resident in the shared system.  There are two
possible configurations for the shared computer system:     A  PDP-10
equivalent  machine  or a software package in ASA Standard Fortran IV
which will run on any sufficiently large machine.    One of the tasks
under this proposal would be to evaluate these two alternatives.

A major share of the programming research effort will be concentrated
on developing a programming system for automated assembly.  This will
combine  ideas  from  our  past  work  on  languages, representation,
computer graphics and arm control.  We view this task as a branch  of
the  larger  effort on automatic programming being undertaken in this
laboratory and elsewhere.